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Why is looking at an 
image on a screen... 
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The virtual window 
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...not like looking 
through a window? 
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The virtual window 
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move our viewpoint and the image looks wrong 
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The virtual window 
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the focus distance is wrong too 
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The virtual window 


The solution? 
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The virtual window 
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Homspaae » Magnifiers » Flexilens on Base, Black 



(^, Magnifying glass 



^ Click on the image to 20 om 



Flexilens on Base, Black 


£39.99 


Quantity [l 


Add to basket 


0€SCRIPTK)N 

This is one oflhe products from the StarMag'^ range The 
hands-free crystal dear large magnifier is ideal for seeing 
detail when crafting, painting or reading It has a very sturdy 
weighted oase and a long reach 46 5cm/ 18* fiewDie arm 
which can oe easily adjustaoie to any position The quality 
13cm/5* large lens provides a 1 75X magnification (3 
diopter) 

• Large magnifying lens for all detailed tasks 

• High quality 13crrV5’ lens with 1 75 X magnification (3 
diopter) 

• Rimless lens for comfortaoie uninterrupted view 

• FlexIOIe 46cm/18* arm for long reach and easy 
adjustment 

• Statue oase for accurate positioning on tables 
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The virtual window 
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.a seamless transition from real to virtual! 
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The virtual window 


projection centre 
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The 3D virtual window (stereoscope) 
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projection centre 




optical centre 
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display 


lens 


a different 
image for 
each eye 





ll 1—^ GAME DEVELOPERS CONFERENCE" EUROPE 2014 GDCEUROPE.COM 

^1^ I -3 AUGUST 11-13, 2014 





The 3D virtual window 
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adjustable for glasses wearers 
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Side-to-side movement 
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tolerant to variations in inter-pupillary distance without adjustment 
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Optics Summary 
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Stereoscopic 3D 
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Ortho-stereoscopic viewing 
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Ortho-stereoscopic viewing 
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Wide field of view optics 




Wide field of view optics 
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Wide field of view optics 





Wide field of view optics 


how to have the 
best of both 
worlds? 
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exploit the eye's 
variable acuity... 
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Wide field of view optics 
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the image with a 
distortion shader 
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Wide field of view optics 




GAME DEVELOPERS CONFERENCE" EUROPE 2014 
AUGUST 11-13.2014 


SONY 


GDCEUROPE.COM 


the optics apply the 
inverse distortion so 
the edges look 
correct again 











Wide field of view optics 
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small pixels in the 
centre, larger pixels 
at the edges 



Tracking 


accelerometers 


gyroscopes 



360° stereoscopic optical tracking 
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Tracking 
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Don't take control of the player's head! 



Take care with first person 
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Take care with first person Abaction 
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Take care with first person action 
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Don't use cinematography! 
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Van 
Filters 
Lens flares 
Bloom 


Film grain 
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Virtual cinema? 
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Stereoscopic Quality checks 
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Stereoscopic Quality checks 
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Stereoscopic Quality checks 
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1. Left and right the correct way round? 

2. Same elements in both eyes? 
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Stereoscopic Quality checks 



1. Left and right the correct way round? 

2. Same elements in both eyes? 
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Stereoscopic Quality checks 
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1. Left and right the correct way round? 

2. Same elements in both eyes? 

3. Both images represent the same time? 
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2. Same elements in both eyes? 

3. Both images represent the same time? 




Stereoscopic Quality checks 
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1 . Left and right the correct way round? 


». Same elements in both eyes? 


3. Both images represent the same time? 


4. The scale is correct? 
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Stereoscopic Quality checks 
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1. Left and right the correct way round? 

2. Same elements in both eyes? 


3. Both images represent the same time? 


4. The scale is correct? 

5. The depth is consistent? 
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Stereoscopic Quality checks 




1. Left and right the correct way round? 

2. Same elements in both eyes? 

3. Both images represent the same time? 


4. The scale is correct? 


5. The depth is consistent? 
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Stereoscopic Quality checks 




1. Left and right the correct way round? 

2. Same elements in both eyes? 

3. Both images represent the same time? 


4. The scale is correct? 


5. The depth is consistent? 
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Stereoscopic Quality checks 
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iTLeftTndrightt^^ way round? 

2. Same elements in both eyes? 

3. Both images represent the same time? 


4. The scale is correct? 


5. The depth is consistent? 

6- ^Pid depth changes avoided? 






GAME DEVELOPERS CONFERENCE" EUROPE 2014 
AUGUST 11-13.2014 


GDCEUROPE.COM 
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Fusion zones 
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Getting The Best 

From Your Next-Gen Engine 
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Overview 

• Every engine is different 

• But there will always be some similarities 

• The considerations I'll discuss today are really important 
for VR... 

• ...but a lot of them are also important for any next-gen 
engine 
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What Makes A Good Engine? 

• High-quality visuals 

• Consistent, high frame rate 

• Low latency 



GAME DEVELOPERS CONFERENCE" EUROPE 2014 
AUGUST 11-13.2014 


GDCEUROPE.COM 



SONY 


What Makes A Good VR Engine? 

• High-quality visuals 

• Consistent, high frame rate 

• Low latency 
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What Makes A Good VR Engine? 

• High-quality visuals 

• Consistent, high frame rate 

• Great tracking and calibration 

• Low latency 
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Testbed Engine 

• I created a very simple testbed engine to test my 
experiments: 

— Stereo 3D rendering 
— Large range of depths 
— Multiple anti-aliasing solutions 
— Geometry optimisations 
— Multi-context rendering 


L 
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High-Quality Visuals 
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High-Quality Visuals 

• What do we mean by high-quality visuals? 

- Nothing distracting you from being immersed in the game 

- Good shading (but not necessarily photorealistic) 

• This often means good anti-aliasing 
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High-Quality Visuals 


• Why is good anti-aliasing essential? 


- The nature of human perception means we're easily distracted 
by high frequency noise 

- Distraction reduces the sense of presence 

- Aliasing artifacts can be even worse with stereo rendering 


• They can cause retinal rivalry 


* Good anti-aliasing can be more important than native 
resolution 
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Anti-Aliasing Methods 

• Edge geometry AA 

- Often hardware accelerated 

• Image-space AA 

— Fit well with most rendering pipelines 

- FXAAiii, MLAAi21, SMAA^^ etc. 

- Temporal AA 

• Use re-projection for temporal supersampling 


• See References at the end of the presentation for more 
info^ 
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Edge-Geometry Anti-Aliasing 
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high-frequency geometry looks better 
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Edge-Geometry Anti-Aliasing 
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a I most-vertical lines look better 



no anti-aliasing 2x MSAA O 
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Edge-Geometry Anti-Aliasing 
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diagonal lines look better 
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Edge-Geometry Anti-Aliasing 
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no anti-aliasing 


2x MSAA 
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Image-Space Anti-Aliasing 
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edge geometry looks better 
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Image-Space Anti-Aliasing 
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texture/shading detail looks better 
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Image-Space Anti-Aliasing 



can sometimes lose detail in high-frequency data 



no anti-aliasing 


FXAA 
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Supersample Anti-Aliasing 

• Good results from rendering to a larger buffer... 

• ...If you can afford it 

• Use with a good downsample filter 



GAME DEVELOPERS CONFERENCE" EUROPE 2014 
AUGUST 11-13.2014 


GDCEUROPE.COM 



Supersample Anti-Aliasing 
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noisy 


much less 
noisy than no 
SSAA 


slightly less 
noisy than 
1.5x 


less noticeable 
difference between this 
and 2x 




Supersample Anti-Aliasing 
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Supersample Anti-Aliasing 
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Stair-Stepped 
edge geometry 



smooth, 
defined edges 



IM 

no SSAA 





it 

1.5x SSAA 
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Supersample Anti-Aliasing 


texture/shading 

aliasing 



no SSAA 
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anti-aliased 

texture/shading 
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Anti-Aliasing Findings 

• Specular AA can also improve the image a lot 

- A great starting point is to look at LEAN, Cheap LEAN (CLEAN) 
and Toksvig AA^^ 

• Distortion shader reduces edge aliasing 

• In some games, more attention may need to be given to 
levels of detail 
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Anti-Aliasing Findings 
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* A combination of several AA methods may give better 
results 

- Each different AA solution combats different aspects of aliasing 

- Use whichever methods work best for your engine 
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High Frame Rate 
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High Frame Rate 

• Why is a consistent, high frame rate essential? 

- Low frame rate looks and feels bad in VR 

— Testing is difficult without a high frame rate 

• Maintain high frame rate throughout development 

- Lack of V-sync is also much more noticeable 

• So make sure V-sync is enabled 
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High Frame Rate 

• The concept of a "pass" is widely-accepted in current engines 

— Reflection rendering, shadow rendering, post-processing etc. 

• Each pass has different requirements 

• Always work out where the bottleneck is for each pass 

- CPU? 

• Draw calls? 

• State setting? 

• Resource setting? 

- GPU? 

• Vertex processing limited? 

• Geometry processing limited? 

• Pixel processing limited? 
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High Frame Rate 

• CPU bound on draw calls, state setting or resource 
setting? 

- Consider how you can utilise the Geometry Shader 

• Can reduce total number of draw calls 

- Shadow cascade render: drawCallCount / n, where n is number of cascades 

- Cube map render: drawCallCount/ 6 

• Reduces resource setting cost 

• It has other features that can help move processing off the CPU 
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High Frame Rate 

• Geometry Shader 

- Converts a stream of primitives into another, possibly larger 
stream of primitives 

- Occurs before the pixel shader 

• i.e. after the vertex shader in a straightforward vertex-pixel draw call 

• After the hull-domain shaders if tessellation is enabled 
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High Frame Rate 

• Geometry Shader Features 

— Render target index/viewport index 

• Use for single-pass cubemap rendering, shadow cascades, S3D etc. 

- GS Instancing 

• Allow multiple executions of the same geometry shader to be run per- 
primitive without the previous shader stage being run again 
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High Frame Rate 

• Geometry Shaders for Stereo 3D Rendering 

- An easy method of making your engine stereoscopic 3D 
compatible 

- Add a GS to every material (or adjust the GS of existing materials 
that already have one) as follows: 
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GS for Stereo 3D Rendering 
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[maxvertexcount (3) ] 
void main ( 

inout Triangles tream<GS_OUTPUT> triangleStream, 
triangle GS_INPUT input [3]) 

{ 

for (uint i = 0; i < 3; ++i) 

{ 

GS_0UTPUT output; 

output .position = (input [i] . worldPosition , g_ViewPro jectionMatrix) ; 
triangleStream . Append (output) ; 

} 

} 




GS for Stereo 3D Rendering 
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[maxvertexcount (3) ] 
void main ( 

inout Triangles tream<GS_OUTPUT> triangleStream, 
triangle GS_INPUT input [3] , 
luint gsinstanceld : SV_GSInstanceID I 

{ 

for (uint i = 0; i < 3; ++i) 

{ 

GS_0UTPUT output; 

, j^^F^wProje^^nMe^r^^gsIn^anceld^ ; 

^i^^^^^enderTargetl^^^^sIn^ancel^J 
triangleStream . Append (output) ; 
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High Frame Rate 


• Vertex/geometry bound? 


- Reduce your vertex size by compressing your attributes 

- Pack any attributes between shader stages 

• Important if geometry shaders for amplification or tessellation pipeline 
are being used 


- Consider using a late fetch method 


• Bind vertex attribute data as a buffer in the shader stage it is used 

• Highly dependent on hardware 

• Always profile to see if it makes a difference! 

- Reduce the data being moved around the GPU 
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High Frame Rate 

• Pixel bound? 

- Reduce the complexity of pixel shaders 

- Reduce the number of pixels shaded every frame 

• An experiment using a smaller render target with upsampling conflicted 
with high-quality visuals, introducing haloing, shimmering and retinal 
rivalry 
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High Frame Rate 

• Consider using re-projection to speed up aspects of 
stereo 3D rendering^ 

- Used with great success on PlayStation®3 stereo 3D games 

- However, it can only be used successfully where parallax is small 



GAME DEVELOPERS CONFERENCE" EUROPE 2014 
AUGUST 11-13.2014 


GDCEUROPE.COM 



Great Tracking and Calibration 
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Great Tracking and Calibration 

• The Project Morpheus SDK handles tracking 

• Use the tracking matrices as supplied by the SDK 
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Tracking Matrices 





existing game 


Default viewing position and 
orientation defined by the game 


y 




Tracking Matrices 
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/ 




existing game 

/ 

y 

Tracked offset of the players head 

1 1 head specific 



from the camera 



Ax B 


Multiplication 
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Tracking Matrices 



existing game 
I I head specific 
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Offset of the player's eyes 
relative to the head matrix 


A X B X C 

Multiplication 
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Tracking Matrices 



existing game 
head specific 




Tracker reset function: 
sets the head position 
to line up with the 
game camera in 
position and yaw. 


A X B X C 

Multiplication 

: o 
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Calibration 
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• Reset positional and orientation tracking 

• Re-align the game world with the real world for: 


- Fixed playing positions 

- Pass-and-play 

- Players of different heights 





Low Latency 
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Latency 

• Why is the reduction of latency so important? 

- Latency is the time interval between an input and the response 

- It isn't just a consistent, high frame rate that's important 

• Not only for VR head-tracking; increasing responsiveness is 
well understood in gaming 

- Gameplay programmers understand the need for responsive controls 

- Network programmers understand the need for responsive opponents 

- etc. 
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Multi-Context Rendering 


• One way to reduce latency from an engine perspective is to 

use deferred contexts to build command lists (AKA command 

buffers) on multiple threads asynchronously 

- As an immediate context "generates rendering overhead when it 
queues up commands in the command buffer. In contrast, a command 
list executes much more efficiently during playback."i§l 




— Works wel l with the concept of a "pass' 

Multi-context rendering reduces latency by allowing the GPU 
to start processing earlier in the frame 

^ O 
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Single-Context Rendering 
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Game Threac^ 

Game Update 


Engine Thread 


Shadow Render 

Main Pass Render Post-Processing t 

■xecute 

GPU 


Execute Commands On GPU 


GPU doesn't start processing work 
until late in the frame 
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Multi-Context Rendering 
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Game Thread 
Engine Thread 0 
Engine Thread 1 
Engine Thread 2 
Engine Thread 3 
GPU 


Game Update 


Shadow Render 


Execute 


Post-Processing 


Shadows 


Left Eye Main Render 

Execute 

Right Eye Main Render 

Execute 


Execute 


Left Eye 


Right Eye 


Post-process 



GPU started earlier 
in the frame, 
so finishes earlier 
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Multi-Context Rendering 

• Simplest test case 

- Built and submitted command lists for each eye's view in parallel 

- Immediate reduction in CPU frame time 

- If the engine is CPU-bound, this translates as an immediate 
reduction in frame latency 

- If the engine is GPU-bound but the GPU is now finishing earlier 
in the frame because it started earlier, this translates as an 
immediate reduction in frame latency 
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Latency Considerations for VR 

• Are there any VR-specific ways to combat latency? 

- The time between sampling the tracking data and using that 
data to render a frame needs to be as low as possible 

- Don't use more than double buffering 

- Re-projecting the image with the latest orientation data can 


improve the apparent latency and frame rate 

• Keep tracked peripherals' latency as low as possible too 

• Are there any platform-specific ways to combat latency? 
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Prediction 

• PlayStation®4 with Project Morpheus is a known system 
— We know any latencies in the hardware 
— We know any latencies in the libraries/software 

- We're continuing to reduce these latencies all the time 

- We give you performance analysis tools for CPU and GPU that 
enable you to calculate and reduce the latency in your game 


• We can use this to predict where the HMD will be by the 
time the image gets displayed 
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Prediction 

• Reducing latency in your engine is key 

- But using prediction to mask any small remaining latencies can 
work well 

- The smaller the amount of prediction you specify, the better its 
quality 
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What's Next? 
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SONY 


What's Next? 

• So now you have great tracking in an incredibly efficient, 
high-frame rate, low-latency, super-high-quality next-gen 
engine that's optimised for virtual reality.. .your job's 
done, isn't it? 

• Of course not! 

- More platform-specific optimisation 

- Tracked peripherals 

- Social aspects 

- Gameplay/design elements 
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Asynchronous Compute 

• Still CPU-bound? 

- Maybe Compute can help offload parallelisable tasks to the GPU 

• Still GPU-bound? 

— Compute allows you to think about GPU tasks from a different, 
more generic perspective 

• Use it where your GPU is not being fully utilised 


- Shadow rendering is usually vertex/geometry-heavy so it's a 
good place to schedule async Compute tasks 
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Tracked Peripherals 


SONY 


• DUALSHOCK®4 

- Motion sensors 

- Light bar 

• PlayStation®Move 

• Multiple devices and 
multiple users 
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Social 


SONY 


• Social Screen 

— Asymmetric gameplay 
— Tracked controllers 

• Companion Apps 

• Online multiplayer 

— Head-tracked opponents 



O 
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Design 

All Access Pass, Student Pass 


Formatil 

Track: 

Pass Type: 


ed Ashforth 
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Senior Game Designer, WWS Immersive Technology Group 
Sony Computer Entertainment Europe 
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Tools and Middleware 


SONY 
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Autodesk 


Scaleform* 


EPICI 

GAMES 


sM. Silicon Studio 
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Thanks 


• Sony Computer Entertainment Europe 

- Simon Benson, Immersive Technology Group 

- Sharwin Raghoebardayal, World Wide Studios 

- Domenico Troiano, Research & Development 
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